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WWC Review of the Report “The Impact of Collaborative Strategic 
Reading on the Reading Comprehension of Grade 5 Students 

in Linguistically Diverse Schools” 1 

The findings from this review do not reflect the full body of research evidence on 

Collaborative Strategic Reading (CSR). 


What is this study about? 

The study examined the impact of Collaborative 
Strategic Reading (CSR), a set of instructional 
strategies used to build reading proficiency, on the 
reading comprehension of fifth-grade students. The 
analysis included 1 ,355 students from 74 social 
studies classrooms within 26 linguistically diverse 
schools. The schools were located in five districts in 
Oklahoma and Texas. 

Within each school, researchers randomly assigned 
social studies classrooms to either the interven- 
tion condition or the comparison condition at the 
start of either the 2006-07 or the 2007-08 school 
year. Teachers in the intervention condition used 
CSR instructional strategies when delivering social 
studies instruction for a period of one school year. 
Teachers in the comparison condition provided 
business-as-usual instruction of social studies. Vari- 
ous curricula were used across both conditions. 

Study authors assessed the effectiveness of CSR 
on reading comprehension by comparing student 
performance on the Group Reading Assessment and 
Diagnostic Evaluation (GRADE) administered in May 
at the end of the school year. 2 


Features of Collaborative Strategic 
Reading (CSR) 


CSR is a set of instructional strategies designed to 
improve the reading comprehension of students with 
diverse abilities. The CSR approach uses both small 
group instruction (groups of 10-12 students) and 
whole class instruction. Teachers introduce CSR 
to students by teaching them four comprehension 
strategies. Then, students apply those strategies 
while working in cooperative learning groups. The 
four strategies include: 

• Preview: scanning text, brainstorming, and 
predicting what will be presented, 

• Click and Clunk: monitoring reading for difficult/ 
unfamiliar words, 

• Get the Gist: identifying the main idea, and 

• Wrap up: summarizing what was read. 
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What did the study find? 

Study authors reported that CSR did not have a sig- 
nificant impact on student reading comprehension, 
and the WWC confirmed this result. In addition, the 
impact of the intervention was not large enough to 
be considered substantively important according to 
WWC criteria. 


WWC Rating 


The research described in this 
report meets WWC evidence 
standards without reservations 

Strengths: The study is a well-implemented 
randomized controlled trial with low attrition 3 at the 
classroom and student levels. 
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Appendix A: Study details 

Hitchcock, J., Dimino, J., Kurki, A., Wilkins, C., & Gersten, R. (2011). The impact of Collaborative Stra- 
tegic Reading on the reading comprehension of grade 5 students in linguistically diverse schools. 
(NCEE 2011-4001). Washington, DC: National Center for Education Evaluation and Regional Assis- 
tance, Institute of Education Sciences, U.S. Department of Education. 


Setting The study was conducted in five school districts in Oklahoma and Texas. The districts were large 
(10,000 or more students) urban or suburban districts with at least 25% of students classified 
as English language learners (ELLs). The participating schools in this study were a convenience 
sample of volunteer schools from these districts, rather than a randomly selected sample. 

Study sample In each of the five districts, social studies classrooms were randomly assigned to either the inter- 
vention or comparison condition within schools. The random assignment process was designed to 
balance the number of classrooms assigned to the conditions within each school and each district. 
Schools had between two and five classrooms that were eligible to participate in the study. There- 
fore, each school had at least one intervention classroom and at least one comparison classroom. 

The initial sample included 86 classrooms from 30 schools. The analysis sample included 1 ,355 
students from 74 classrooms in 26 schools. This included 227 students that had either pretest 
or posttest data imputed by the authors. Imputation was conducted only for students who were 
enrolled at baseline, had parental consent, were testable, and had at least one valid Group Read- 
ing Assessment and Diagnostic Evaluation test score at either pretest or posttest. 4 

Approximately 48% of the analysis sample were male, 63% were Hispanic, 24% were African 
American, and 73% qualified for free or reduced-price lunch. In addition, 44% of the students 
were currently or previously classified as ELLs. 


Intervention Teachers in the intervention group used the Collaborative Strategic Reading approach while 
group providing social studies instruction using various curricula for a period of one year. 

CSR is a set of instructional strategies designed to improve the reading comprehension of 
students with diverse abilities. The CSR approach uses both small group instruction (groups of 
10-12 students) and whole class instruction. Teachers introduce CSR to students by teaching 
them four comprehension strategies. Then, students apply those strategies while working in 
cooperative learning groups. The four strategies include: 

• Preview: scanning text, brainstorming, and predicting what will be presented, 

• Click and Clunk: monitoring reading for difficult/unfamiliar words, 

• Get the Gist: identifying the main idea, and 

• Wrap up: summarizing what was read. 

In the intervention classrooms, observations were conducted to assess the fidelity of implementa- 
tion of CSR strategies. Observations focused on the degree to which teachers were (a) conducting 
a whole class preview to introduce the lesson, (b) pre-teaching vocabulary, (c) circulating among 
groups and monitoring students’ use of strategies during group work, (d) circulating among groups 
and reinforcing content learning during group work, and (e) providing ongoing instruction in the 
comprehension strategies (e.g., modeling or coaching on how to use strategy). Observational data 
indicated that 21 .6% of teachers used all five of these strategies, while 56.7% used three or fewer. 
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Comparison 

group 

Outcomes and 
measurement 


Support for 
implementation 


Reason for 
review 


Teachers in the comparison condition provided business-as-usual social studies instruction 
without the CSR strategies being explicitly used. 

The study examined scores on the GRADE assessment in the spring of the school year dur- 
ing which the intervention was implemented. For a more detailed description of this outcome 
measure, see Appendix B. 

Each district held a 2-day training session for all teachers at the start of the school year, prior 
to pretesting. Both intervention and comparison teachers attended the introductory overview 
session. Intervention teachers then participated in four CSR training segments: theoretical 
foundations of CSR, CSR comprehension strategies, implementation practices and logistics, 
and coaching. The training was conducted by the study’s co-principal investigator and the co- 
developer of CSR and was also attended by coaches assigned to the intervention teachers. 
Ninety-two percent of teachers in the intervention group attended both days of training. Those 
who could not attend due to medical reasons received a DVD recording of the training and a 
phone call from their assigned coach. In addition, intervention teachers received four follow-up 
coaching sessions during the school year. 

This study was identified for review by the WWC because it is an Institute of Education Sci- 
ences (lES)-funded study conducted by 2006-1 1 Regional Education Laboratory Southwest at 
Edvance Research. 
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Appendix B: Outcome measure for the reading comprehension domain 


Reading comprehension 


Group Reading Assessment and GRADE is a standardized, nationally normed, group-administered assessment of student reading, including 

Diagnostic Evaluation vocabulary, reading comprehension, and metacognitive skills for understanding informational text. The total 

scaled score includes subscales for vocabulary, sentence comprehension, and passage comprehension. Form 
A was administered as the pretest in the fall of the intervention year and Form B as the posttest in the spring 
of the intervention year; the two forms are equivalent. The test is untimed, and a second session was provided 
for struggling readers when determined to be appropriate by the lead data collector and classroom teachers (as 
recommended by the test developer). 


July 2013 


Page 5 



WWC Single Study Review 


Appendix C: Study findings for the reading comprehension domain 


Mean 

(standard deviation) WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Reading comprehension 

GRADE 

CSRv s. 
comparison 

74 classrooms/ 
1,355 students 

98.67 

(12.06) 

98.01 

(11.71) 

0.66 

0.05 

+2 

0.11 


Domain average for reading comprehension 0.05 +2 Not 

statistically 

significant 

Table Notes: Positive results for mean difference, effect size, and improvement index favor the intervention group; negative results favor the comparison group. The effect size is 
a standardized measure of the effect of an intervention on student outcomes, representing the change (measured in standard deviations) in an average student’s outcome that can 
be expected if the student is given the intervention. The improvement index is an alternate presentation of the effect size, reflecting the change in an average student's percentile 
rank that can be expected if the student is given the intervention. GRADE = Group Reading Assessment and Diagnostic Evaluation. 

Study Notes: The sample sizes, means, standard deviations, mean difference, effect size, and the p-value presented in this report were taken from impact estimates reported in 
the study. The authors estimated impacts using two-level hierarchical linear modeling (HLM) that nested students within classrooms and estimated an intervention effect for each 
school. The overall adjusted mean difference was then calculated as an average of the school-level impacts weighted by the number of students at each school. Due to missing 
student-level outcomes at pretest or posttest, a multiple imputation procedure was used and implemented in imputation and variance estimation software (IVEware) (Raghunathan 
et al., 2001). All estimates, including p-values, were adjusted for the imputation process and satisfy WWC criteria. The statistical significance of the study's domain average was 
determined by the WWC; the study is characterized as having no discernible effects on reading comprehension because the effect on one outcome examined was not statistically 
significant or substantively important. 
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Endnotes 

1 Single study reviews examine evidence published in a study (supplemented, if necessary, by information obtained directly from the 
authors]) to assess whether the study design meets WWC evidence standards. The review reports the WWC’s assessment of whether 
the study meets WWC evidence standards and summarizes the study findings following WWC conventions for reporting evidence on 
effectiveness. This study was reviewed using the Adolescent Literacy review protocol, version 2.0. 

2 The authors also assess the effectiveness of CSR for two subgroups: all former and current English language learner (ELL) students 
and non-ELL students. Those findings are not reported in this review for two reasons: (a) insufficient information is reported in the 
study to apply WWC evidence standards to the subgroup analyses, and (b) the authors clearly note that the analysis of these sub- 
groups was for the purpose of an exploratory exercise. 

3 Two research decisions influenced the assessment of attrition results reported in this single study review: (a) random assignment 
was performed before families provided consent, and (b) multiple imputation was used to produce a complete dataset without miss- 
ing data for the final analysis sample. To proceed, the WWC assessed sub-cluster attrition using the following student samples: (a) at 
baseline, the student sample was taken as those in classrooms at the time of random assignment (intervention = 827, comparison = 
825), regardless of parental consent, and (b) at analysis, the WWC used students who completed both pre- and posttests (intervention 
= 584, comparison = 544), that is, without imputation, as the analytic sample. Based on this most conservative assessment, attrition is 
low according to WWC standards. 

4 All WWC criteria for implementing an acceptable multiple imputation method are satisfied in this study. First, attrition in the absence 
of imputation is low. Second, the authors name the method and software that was used to implement the method: Raghunathan et al. 
(2001) and IVEware (citations below). Third, the authors perform the imputation separately by intervention condition. Fourth, more than 
one imputed dataset was created for the analysis— in this case, ten. Finally, all analyses, including inference, were adjusted according 
to the multiply imputed datasets. 

Raghunathan, T. E., Lepkowski, J. M., Van Hoewyk, J., & Solenberger, P. (2001). A multivariate technique for multiple imputing 
missing values using a sequence of regression models. Survey Methodology, 27(1), 85-95. 

Raghunathan, T. E., Solenberger, P. W., & Van Howeyk, J. (2002). IVEware: Imputation and variance estimation 
software [computer software] . Ann Arbor, Ml: University of Michigan. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2013, July). 

WWC review of the report: The impact of Collaborative Strategic Reading on the reading comprehension 
of grade 5 students in linguistically diverse schools. Retrieved from http://whatworks.ed.gov 
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Glossary of Terms 

Attrition 


Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Improvement index 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Single-case design 
(SCD) 

Standard deviation 


Statistical significance 
Substantively important 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review if it falls within the scope of the review protocol and uses either 
an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into intervention and comparison groups. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample are spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < 0.05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 2.1) for additional details. 
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